Vehicle lidar system with neural network-based dual density point cloud generator

ABSTRACT

A vehicle system includes a lidar system that obtains an initial point cloud and obtains a dual density point cloud by implementing a first neural network and based on the initial point cloud. The dual density point cloud results from reducing point density of the initial point cloud outside a region of interest (ROI). Processing the dual density point cloud results in a detection result that indicates any objects in a field of view (FOV) of the lidar system. A controller obtains the detection result from the lidar system and controls an operation of the vehicle based on the detection result.

INTRODUCTION

The subject disclosure relates to a vehicle lidar system with a neuralnetwork-based dual density point cloud generator.

Vehicles (e.g., automobiles, trucks, construction equipment, farmequipment) increasingly rely on sensors to provide information about thevehicle and its environment. Exemplary types of sensors that provideinformation about the environment around the vehicle include a radiodetection and ranging (radar) system, a light detection and ranging(lidar) system, and a camera. A lidar system provides a point cloudrepresentation of features in the field of view of the lidar system. Thedetection of objects in the field of view is improved with an increaseddensity of the point cloud but the processing time and complexity arealso increased with density. Accordingly, it is desirable to provide avehicle lidar system with a neural network-based dual density pointcloud generator.

SUMMARY

In one exemplary embodiment, a vehicle system includes a lidar system toobtain an initial point cloud and to obtain a dual density point cloudby implementing a first neural network and based on the initial pointcloud. The dual density point cloud results from reducing point densityof the initial point cloud outside a region of interest (ROI).Processing the dual density point cloud results in a detection resultthat indicates any objects in a field of view (FOV) of the lidar system.A controller obtains the detection result from the lidar system andcontrols an operation of a vehicle based on the detection result.

In addition to one or more of the features described herein, the lidarsystem implements the first neural network to define the ROI within theFOV that results in the dual density point cloud, the ROI being a regionof fixed area with a center that is selected from a set of potentialcenters based on an output of the first neural network.

In addition to one or more of the features described herein, the lidarsystem implements a second neural network to output the detection resultbased on the dual density point cloud.

In addition to one or more of the features described herein, the secondneural network includes an encoder and decoder stage that providespoint-wise feature vectors such that each feature vector of thepoint-wise feature vectors is associated respectively with each point ofthe dual density point cloud.

In addition to one or more of the features described herein, the firstneural network is a Deep Q-Network (DQN) that obtains the point-wisefeature vectors from the encoder and decoder stage of the second neuralnetwork.

In addition to one or more of the features described herein, trainingthe DQN includes comparing the detection result obtained with the dualdensity point cloud with a ground truth detection result to produce anumber of true positives and comparing a second detection resultobtained by reducing the point density of the initial point cloudthroughout the FOV with the ground truth detection result to produce asecond number of true positives.

In addition to one or more of the features described herein, thetraining the DQN includes obtaining a reward by comparing the number oftrue positives with the second number of true positives, and thetraining the DQN includes maximizing the reward.

In addition to one or more of the features described herein, the DQNoutputs a matrix indicating a predicted reward corresponding with eachpotential center among the set of potential centers.

In addition to one or more of the features described herein, the DQNoutputs a matrix indicating a probability of a positive rewardcorresponding with each potential center among the set of potentialcenters.

In addition to one or more of the features described herein, thetraining the DQN includes obtaining a loss as a difference between thereward and a predicted reward provided by the DQN, and the training theDQN includes minimizing the loss.

In another exemplary embodiment, a method includes obtaining an initialpoint cloud and implementing a first neural network to obtain a dualdensity point cloud based on the initial point cloud. The dual densitypoint cloud results from reducing point density of the initial pointcloud outside a region of interest (ROI). The method also includesprocessing the dual density point cloud to obtain a detection resultthat indicates any objects in a field of view (FOV) of the lidar system.

In addition to one or more of the features described herein, theimplementing the first neural network results in defining the ROI withinthe FOV that results in the dual density point cloud, the ROI being aregion of fixed area with a center that is selected from a set ofpotential centers based on an output of the first neural network.

In addition to one or more of the features described herein, the methodalso includes implementing a second neural network to output thedetection result based on the dual density point cloud.

In addition to one or more of the features described herein, theimplementing the second neural network includes implementing an encoderand decoder stage to provide point-wise feature vectors such that eachfeature vector of the point-wise feature vectors is associatedrespectively with each point of the dual density point cloud.

In addition to one or more of the features described herein, the firstneural network is a Deep Q-Network (DQN) and implementing the DQNincludes obtaining the point-wise feature vectors from the encoder anddecoder stage of the second neural network.

In addition to one or more of the features described herein, the methodalso includes training the DQN based on comparing the detection resultobtained with the dual density point cloud with a ground truth detectionresult to produce a number of true positives and comparing a seconddetection result obtained by reducing the point density of the initialpoint cloud throughout the FOV with the ground truth detection result toproduce a second number of true positives.

In addition to one or more of the features described herein, thetraining the DQN includes obtaining a reward by comparing the number oftrue positives with the second number of true positives, and thetraining the DQN includes maximizing the reward, and the training theDQN additionally includes obtaining a loss as a difference between thereward and a predicted reward provided by the DQN, and the training theDQN includes minimizing the loss.

In addition to one or more of the features described herein, theimplementing the DQN includes outputting a matrix indicating a predictedreward corresponding with each potential center among the set ofpotential centers.

In addition to one or more of the features described herein, theimplementing the DQN includes outputting a matrix indicating aprobability of a positive reward corresponding with each potentialcenter among the set of potential centers.

In addition to one or more of the features described herein, the methodalso includes a vehicle controller obtaining the detection result fromthe lidar system and controlling an operation of a vehicle based on thedetection result.

The above features and advantages, and other features and advantages ofthe disclosure are readily apparent from the following detaileddescription when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, advantages and details appear, by way of example only,in the following detailed description, the detailed descriptionreferring to the drawings in which:

FIG. 1 is a block diagram of a vehicle with a neural network-based dualdensity point cloud generator according to one or more embodiments;

FIG. 2 illustrates an exemplary field of view with a neuralnetwork-based dual density point cloud according to one or moreembodiments;

FIG. 3 is a process flow of a method of performing a neuralnetwork-based dual density point cloud generation in a lidar system of avehicle according to one or more embodiments; and

FIG. 4 is a process flow of aspects of a method of training the neuralnetwork that is implemented in the process flow shown in FIG. 3 .

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, its application or uses. Itshould be understood that throughout the drawings, correspondingreference numerals indicate like or corresponding parts and features.

As previously noted, a lidar system is one of the sensors used to obtaininformation about an environment around a vehicle. As also noted, thedensity of the point cloud obtained from the lidar system affectsperformance and processing time. Specifically, performance increaseswith an increase in the density of the point cloud, but so doesprocessing time and bandwidth cost. Embodiments of the systems andmethods detailed herein relate to a vehicle lidar system with a neuralnetwork-based dual density point cloud generator. The neural network ispart of the lidar system rather than one that performs post-processingon an output point cloud from the lidar system.

A dual density point cloud refers to a point cloud with an area of lowerdensity points and an area of higher density points. Specifically, theinitially output point density is only retained for a portion of thefield of view (FOV) that is of interest (i.e., the region of interest(ROI)). In other parts of the FOV, the density is decreased to apredetermined percentage. The predetermined percentage of original pointdensity in non-ROI regions of the FOV is based on a recognition thatdetection performance does not increase, even though processing timedoes increase, by retaining points beyond the predetermined percentageof point density in the non-ROI region. The dual density approachfacilitates having the higher processing time and bandwidthcorresponding with higher performance only in the ROI and, thereby,decreasing the overall processing time and bandwidth requirement for thelidar system. For example, to emulate human vision, lower point clouddensity (i.e., lower resolution) may be provided in peripheral areas ofthe field of view. As detailed, the size of the area of the ROI, inwhich point cloud density is relatively higher than in other areas ofthe FOV, may be fixed. The neural network is used to determine where,within the FOV, the ROI should be centered.

In accordance with an exemplary embodiment, FIG. 1 is a block diagram ofa vehicle 100 with a neural network-based dual density point cloudgenerator. The exemplary vehicle 100 shown in FIG. 1 is an automobile101. The vehicle 100 includes a lidar system 110 and may also includeother sensors 130 (e.g., radar system, camera). The number and locationsof the lidar system 110 and the number of other sensors 130 and theirlocations are not limited by the exemplary illustration in FIG. 1 . Thelidar system 110 includes a controller 120 that implements the neuralnetwork-based dual density point cloud generator according to one ormore embodiments.

Specifically, the controller 120 of the lidar system 110 determines thelocation of the ROI 220 within the FOV 210, as shown in FIG. 2 , byimplementing a neural network. As previously noted, the higher densitypoint cloud is maintained only within the ROI 220 and the dual densitypoint cloud is generated by reducing the point density in areas of theFOV 210 outside the ROI 220. According to different exemplaryembodiments, the neural network implemented by the controller 120outputs different indicators of the ROI 220 location within the FOV 210.

The vehicle 100 includes a vehicle controller 140 that may obtaininformation from the lidar system 110 and other sensors 130 to controlan aspect of autonomous or semi-autonomous operation of the vehicle 100.For example, semi-autonomous operation such as adaptive cruise controlor automatic braking may be implemented by the vehicle controller 140based on information from the lidar system 110 and/or other sensors 130.The controller 120 of the lidar system 110 and the vehicle controller140 may both include processing circuitry that may include anapplication specific integrated circuit (ASIC), an electronic circuit, aprocessor (shared, dedicated, or group) and memory that executes one ormore software or firmware programs, a combinational logic circuit,and/or other suitable components that provide the describedfunctionality.

FIG. 2 illustrates an exemplary FOV 210 with a neural network-based dualdensity point cloud according to one or more embodiments. The FOV 210 ofthe lidar system 110 is shown discretized to a grid in FIG. 2 . Asindicated, a horizontal span of the FOV 210 (HFOV) is 90 degrees (deg)and a vertical span of the FOV 210 (VFOV) is 50 deg. An exemplary ROI220 is shown in FIG. 2 . Based on the exemplary grid, the horizontalgrid span of the ROI 220 (Rh) is 9 grid units and a vertical grid spanof the ROI 220 (Rv) is 5 grid units. The center 235 of the exemplary ROI220 is labeled. The center 235 is part of a set of potential centers230, which is a subset of all the units of the grid representing the FOV210 that could act as the center 235 of the ROI 220 based on the fixedsize (i.e., Rh and Rv) of the ROI 220.

That is, selecting a different center 235 from among the set ofpotential centers 230 would result in a shift of the ROI 220 within theFOV 210. The units of the grid representation of the full FOV 210 thatmake up the set of potential centers 230 is limited by the fixed area ofthe ROI 220. That is, the set of potential centers 230 is selected suchthat an ROI 220 centered at any one of the set of potential centers 230will not fall outside the FOV 210. According to one or more embodimentsand as detailed herein, a neural network is implemented by thecontroller 120 of the lidar system 110 to select the center 235 fromamong the set of potential centers 230 and thereby define a location ofthe ROI 220 within the FOV 210.

FIG. 3 is a process flow of a method 300 of performing a neuralnetwork-based dual density point cloud generation in a lidar system 110of a vehicle 100 according to one or more embodiments. At block 310, theprocesses include obtaining the initial point cloud P_(t) that isgenerated within the lidar system 110. The initial point cloud P_(t) hasa uniform density of points throughout the FOV 210. At block 320,reducing the density of the initial point cloud P_(t) in areas outsidethe ROI 220 produces a dual density point cloud {hacek over (P)}_(t).The percentage by which point density is reduced in areas of the FOV 210that are outside the ROI 220 may be fixed (e.g., 20 percent). This dualdensity point cloud {hacek over (P)}_(t) is further processed using aneural network according to known processes (at blocks 330-360) toobtain a detection result D that indicates detected objects and lanesaround the vehicle 100. The neural network may be a region-basedconvolutional neural network (R-CNN), for example.

At block 330, implementing the encoder/decoder stage of the neuralnetwork results in the dual density point cloud {hacek over (P)}_(t)points being mapped to lower-level representations. The decoder layersthen perform up-sampling and generate point-wise feature vectors X_(t).The point-wise vectors refers to the fact that a vector is generated perpoint of the dual density point cloud {hacek over (P)}_(t) points. Forexample, for each of N points in the dual density point cloud {hacekover (P)}_(t), an N×M matrix may be generated or, put another way, anM-length vector (e.g., M=128) may generated for each point as part ofthe point-wise feature vectors X_(t). At block 340, generatingthree-dimensional proposals refers to the fact that each point isclassified as a foreground point or a background point. At block 340, athree-dimensional region is generated as a proposal for an objectassociated with each foreground point. At block 350, processes performedby the neural network include pooling the point cloud regions. Regionpooling refers to combining the three-dimensional region proposals thatcorrespond to the same object. At block 360, refining three-dimensionalbounding boxes results in the detection of objects and lanes in the FOV210. The detection result D from block 360 may be provided to thevehicle controller 140 to affect an operation of the vehicle 100.

At block 370, implementing the other neural network refers toimplementing a Deep Q-Network (DQN). The point-wise feature vectorsX_(t) from the encoder/decoder (at block 330) are also provided to theDQN, as shown in FIG. 3 . The DQN estimates a Q value for each possibleaction a based on weights θ. The result is a matrix A_(t) whose size isthe same as the unit size of the set of potential centers 230 (e.g.,19-by-11 according to the exemplary case shown in FIG. 2 ). That is,each action a is the selection of one of the set of potential centers230 as the center 235 of the ROI 220. At block 380, determining the ROI220 refers to determining the center 235 of the ROI 220 for the nextpoint cloud P_(t+1) output by the lidar system 110 at block 310 for thenext frame.

According to an exemplary embodiment, the matrix A_(t), output fromblock 370, includes a predicted reward associated with each positionwithin the set of potential centers 230. In this case, determining ROI220, at block 380, involves determining which of the set of potentialcenters 230 is associated with the highest predicted reward, accordingto the matrix A_(t). According to another exemplary embodiment, thematrix A_(t), output from block 370, includes a probability associatedwith each position within the set of potential centers 230. In thiscase, determining ROI 220 for the next point cloud P_(t+1), at block380, involves determining which of the set of potential centers 230 isassociated with the highest probability of producing a positive reward,according to the matrix A_(t) According to this embodiment, referred toas a policy gradient, the DQN implements an additional softmax layer toobtain the probability of producing a positive reward. The reward isfurther discussed with reference to FIG. 4 .

According to an exemplary embodiment, the DQN implemented at block 370may be simplified by splitting the x and y dimensions. That is, insteadof one Q value for each grid point in the FOV 210 that may act as thecenter 235 of the ROI 220, a Qx and a Qy may separately be determined bytwo branches of DQN at block 370. Then an Ax_(t) and an Ay_(t) may beoutput by the DQN.

FIG. 4 is a process flow of aspects of a method 400 of training the DQNthat is implemented at block 370 of FIG. 3 . Training the DQN that isimplemented at block 370 involves comparing the detection result D(output by block 360) that is obtained using the ROI 220 (at block 320)with ground truth and also comparing a detection result D′ that wouldhave been obtained if no dual density point cloud {hacek over (P)}_(t)were used (i.e., with density of the initial point cloud P_(t) reduceduniformly over the entire FOV 210) with ground truth. That is, theimprovement in the detection result D that is obtained using the dualdensity point cloud {hacek over (P)}_(t) over a detection result D′ thatis obtained without maintaining a higher point density in the ROI 220 isused to train the DQN.

At block 410, obtaining the detection result D at the output of block360 is detailed with reference to FIG. 3 . At block 420, obtaining thedetection result D′ involves using the same R-CNN discussed withreference to FIG. 3 . However, instead of the input to the R-CNN(implemented at blocks 330-360) being the dual density point cloud{hacek over (P)}_(t), the input is the initial point cloud P_(t) withdensity reduced uniformly over the entire FOV 210. At block 430,comparing the detection result D with ground truth yields a number oftrue positives TP. The true positives refer to the number of detectedobjects in the detection result D that match ground truth. At block 430comparing the detection result D′ with ground truth yields a number oftrue positives TP′ in the detection result D′.

At block 440, comparing the true positives TP obtained using the dualdensity point cloud {hacek over (P)}_(t) with the true positives TP′obtained using the uniformly reduced point density provides the rewardfor the DQN. For example, if the true positives TP exceed the truepositives TP′ (i.e., the dual density point cloud {hacek over (P)}_(t)yielded a more accurate detection result D), then the reward may be apositive value. If the true positives TP equal the true positives TP′(i.e., the dual density point cloud {hacek over (P)}_(t) yielded thesame accuracy as the uniformly reduced point density), then the rewardmay be zero. If the true positives TP are less than the true positivesTP′ (i.e., the dual density point cloud {hacek over (P)}_(t) yielded aless accurate detection result D than using the uniformly reduced pointdensity), then the reward may be a negative value. Training the DQN tomaximize the reward is referred to as reinforcement learning.

This is the reward discussed with reference to the output of block 370.As previously noted, the output of block 370 may be the reward predictedbased on each of the set of potential centers 230 being selected as thecenter 235 of the ROI 220 according to one exemplary embodiment.According to another exemplary (policy gradient) embodiment, the outputof block 370 may be a probability that the reward is a positive valuebased on each of the set of potential centers 230 being selected as thecenter 235 of the ROI 220.

In addition to the reward, a loss may be used in training the DQN. Theloss results from a comparison of predicted reward and actual reward.Thus, rather than using detection result D and detection result D′ (asdiscussed for determination of reward), predicted reward at the outputof the DQN is compared with actual reward. The larger the differencebetween predicted reward and actual reward, the larger the lossattributed to the DQN during training. Thus, the training process seeksto minimize the loss in addition to maximizing the reward.

While the above disclosure has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from its scope. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the disclosure without departing from the essentialscope thereof. Therefore, it is intended that the present disclosure notbe limited to the particular embodiments disclosed, but will include allembodiments falling within the scope thereof

What is claimed is:
 1. A vehicle system comprising: a lidar systemconfigured to obtain an initial point cloud, to obtain a dual densitypoint cloud based on a first neural network and based on the initialpoint cloud, wherein the dual density point cloud results from reducingpoint density of the initial point cloud outside a region of interest(ROI), and to process the dual density point cloud to obtain a detectionresult that indicates any objects in a field of view (FOV) of the lidarsystem; wherein the first neural network is a Deep Q-Network (DQN) thatobtains the point-wise feature vectors from an encoder and decoder stageof a second neural network, and wherein training the DQN includescomparing the detection result obtained with the dual density pointcloud with a ground truth detection result to produce a number of truepositives and comparing a second detection result obtained by reducingthe point density of the initial point cloud throughout the FOV with theground truth detection result to produce a second number of truepositives; and a controller configured to obtain the detection resultfrom the lidar system and to control an operation of a vehicle based onthe detection result.
 2. The vehicle system according to claim 1,wherein the lidar system is configured to implement the first neuralnetwork to define the ROI within the FOV that results in the dualdensity point cloud, the ROI being a region of fixed area with a centerthat is selected from a set of potential centers based on an output ofthe first neural network.
 3. The vehicle system according to claim 2,wherein the lidar system is configured to implement a second neuralnetwork to output the detection result based on the dual density pointcloud.
 4. The vehicle system according to claim 3, wherein the secondneural network includes an encoder and decoder stage configured toprovide point-wise feature vectors such that each feature vector of thepoint-wise feature vectors is associated respectively with each point ofthe dual density point cloud.
 5. A vehicle system comprising: a lidarsystem configured to obtain an initial point cloud, to obtain a dualdensity point cloud based on a first neural network and based on theinitial point cloud, wherein the dual density point cloud results fromreducing point density of the initial point cloud outside a region ofinterest (ROI), and to process the dual density point cloud to obtain adetection result that indicates any objects in a field of view (FOV) ofthe lidar system; the lidar system being configured to implement thefirst neural network to define the ROI within the FOV that results inthe dual density point cloud, and configured to implement a secondneural network to output the detection result based on the dual densitypoint cloud, the second neural network including an encoder and decoderstage configured to provide point-wise feature vectors such that eachfeature vector of the point-wise feature vectors is associatedrespectively with each point of the dual density point cloud; the ROIbeing a region of fixed area with a center that is selected from a setof potential centers based on an output of the first neural network;wherein the first neural network is a Deep Q-Network (DQN) that obtainsthe point-wise feature vectors from the encoder and decoder stage of thesecond neural network, and wherein training the DQN includes comparingthe detection result obtained with the dual density point cloud with aground truth detection result to produce a number of true positives andcomparing a second detection result obtained by reducing the pointdensity of the initial point cloud throughout the FOV with the groundtruth detection result to produce a second number of true positives; anda controller configured to obtain the detection result from the lidarsystem and to control an operation of a vehicle based on the detectionresult.
 6. The vehicle system according to claim 5, wherein the trainingthe DQN includes obtaining a reward by comparing the number of truepositives with the second number of true positives, and the training theDQN includes maximizing the reward.
 7. The vehicle system according toclaim 6, wherein the DQN is configured to output a matrix indicating apredicted reward corresponding with each potential center among the setof potential centers.
 8. The vehicle system according to claim 6,wherein the DQN is configured to output a matrix indicating aprobability of a positive reward corresponding with each potentialcenter among the set of potential centers.
 9. The vehicle systemaccording to claim 6, wherein the training the DQN includes obtaining aloss as a difference between the reward and a predicted reward providedby the DQN, and the training the DQN includes minimizing the loss.
 10. Amethod comprising: obtaining, using a lidar system, an initial pointcloud; implementing a first neural network, using the lidar system, toobtain a dual density point cloud based on the-first neural network andbased on the initial point cloud, wherein the dual density point cloudresults from reducing point density of the initial point cloud outside aregion of interest (ROI), wherein implementing the first neural networkresults in defining the ROI within the FOV that results in the dualdensity point cloud, the ROI being a region of fixed area with a centerthat is selected from a set of potential centers based on an output ofthe first neural network, wherein the first neural network is a DeepQ-Network (DQN) and implementing the DQN includes obtaining point-wisefeature vectors from an encoder and decoder stage of a second neuralnetwork; implementing the second neural network to output the detectionresult based on the dual density point cloud, wherein the implementingthe second neural network includes implementing the encoder and decoderstage to provide point-wise feature vectors such that each featurevector of the point-wise feature vectors is associated respectively witheach point of the dual density point cloud; training the DQN based oncomparing the detection result obtained with the dual density pointcloud with a ground truth detection result to produce a number of truepositives and comparing a second detection result obtained by reducingthe point density of the initial point cloud throughout the FOV with theground truth detection result to produce a second number of truepositives; and processing the dual density point cloud to obtain adetection result that indicates any objects in a field of view (FOV) ofthe lidar system.
 11. The method according to claim 10, wherein thetraining the DQN includes obtaining a reward by comparing the number oftrue positives with the second number of true positives, and thetraining the DQN includes maximizing the reward, and the training theDQN additionally includes obtaining a loss as a difference between thereward and a predicted reward provided by the DQN, and the training theDQN includes minimizing the loss.
 12. The method according to claim 11,wherein the implementing the DQN includes outputting a matrix indicatinga predicted reward corresponding with each potential center among theset of potential centers.
 13. The method according to claim 11, whereinthe implementing the DQN includes outputting a matrix indicating aprobability of a positive reward corresponding with each potentialcenter among the set of potential centers.
 14. The method according toclaim 11, further comprising a vehicle controller obtaining thedetection result from the lidar system and controlling an operation of avehicle based on the detection result.